Simultaneous gesture and touch control on a display

ABSTRACT

A method of controlling contents displayed on a display includes processing inputs from different users across multiple application windows of one or more applications displayed on a single display of a user interface, resulting in a multi-user graphical user interface (GUI), the computing environment running an operating system (OS), the computing environment having a multiple input detection system to respond to user inputs from different users simultaneously, the multiple input detection system including a touch screen on the single display that detects touch inputs and a contactless gesture sensor that detects contactless gestures. When a user input is detected by the touch detector, the GUI displays a first input interface to be used with touch. When a user input is detected by the contactless gesture sensor, the GUI displays a second input interface to be used with gestures, the second input interface being different from the first input interface.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a divisional of U.S. application Ser. No.16/589,648, filed Oct. 1, 2019, which claims priority under 35 U.S.C. §119(e) to U.S. Provisional Application No. 62/739,619, filed on Oct. 1,2018, and entitled: “Simultaneous Gesture and Touch Control on aDisplay,” each of which is incorporated herein by reference in itsentirety.

DESCRIPTION OF RELATED ART

There are currently different solutions to using gestures to control adisplay remotely and different solutions to using touch control on adisplay when in contact with the display. Gesture solutions includesensing gestures using a wearable device and using complex trajectoryinformation. However, using both types of inputs simultaneously has notbeen realized.

SUMMARY

In a computing environment, a method for processing inputs fromdifferent users across multiple application windows of one or moreapplications displayed on a single display of a user interface,resulting in a multi-user graphical user interface (GUI), the computingenvironment running an operating system (OS), the computing environmenthaving a multiple input detection system to respond to user inputs fromdifferent users at a same time, the multiple input detection systemincluding a touch screen on the single display that detects touch inputsand a contactless gesture sensor that detects contactless gestures, themethod including receiving a first user input from the touch screen;simultaneous to receiving the first user input from the touch screen,receiving a different second user input from the contactless gesturesensor; determining, in a primary application, separate from the OS,that the first user input corresponds to a first window; the primaryapplication sending the first user input to a first secondaryapplication that runs in parallel with the primary application, thefirst secondary application being designed to run with the OS;determining a first user input event from the first user input;displaying a result of the determined first user input event in thefirst window associated with the first secondary application;determining, in the primary application, that a second user inputcorresponds to a second window; the primary application sending thesecond user input to a second secondary application that runs inparallel with the primary application, the second secondary applicationbeing designed to run with the OS; determining a second user input eventfrom the second user input; and displaying a result of the determinedsecond user input event on the second window associated with the secondsecondary application.

In a computing environment, a method of controlling contents displayedon a display, method for processing inputs from different users acrossmultiple application windows of one or more applications displayed on asingle display of a user interface, resulting in a multi-user graphicaluser interface (GUI), the computing environment running an operatingsystem (OS), the computing environment having a multiple input detectionsystem to respond to user inputs from different users at a same time,the multiple input detection system including a touch screen on thesingle display that detects touch inputs and a contactless gesturesensor that detects contactless gestures, the method including when auser input is detected by the touch screen, the GUI displays a firstinput interface to be used with touch; and when a user input is detectedby the contactless gesture sensor, the GUI displays a second inputinterface to be used with gestures, the second input interface beingdifferent from the first input interface.

The first input interface may be in a periphery of a selected window andthe second input interface overlaps a selected window.

The second input interface may include at least two buttons.

An area of the second input interface outside the at least two buttonsmay have a default operation.

The second input interface may include another input interface outsideof the selected window.

The second input interface may include a mode selection.

The first input interface may include a first set of operations and thesecond input interface includes a second set of operations, differentfrom the first set of operations.

The second set of operations may have fewer operations than the firstset of operations.

The second set of operations may be a subset of the first set ofoperations.

The second set of operations may include an operation not in the firstset of operations.

When an open hand gesture is sensed, an icon is displayed on thedisplay, and moving an open hand over the icon may move the icon on thedisplay in accordance with the movement of the open hand and a closedhand over the icon may move a virtual canvas on the display inaccordance with the movement of the closed hand.

When a closed hand gesture is sensed, a window overlapped by the closedhand may be selected.

When an open hand gesture is sensed on a window, buttons correspondingto the second input interface may be displayed on the window overlappedby the open hand.

When a closed hand gesture is sensed on a button, the operations of thesecond input interface that button corresponds to may be executed.

The contactless gesture sensor may include a camera separate from avideo-conferencing camera associated with the display.

The OS may be designed to run with a single user GUI.

BRIEF DESCRIPTION OF THE DRAWINGS

Features will become apparent to those of skill in the art by describingin detail exemplary embodiments with reference to the attached drawingsin which:

FIG. 1 illustrates a block diagram of a display system in accordancewith embodiments;

FIGS. 2A to 2C illustrate types of hand gestures to be sensed;

FIGS. 3 to 13 illustrate different actions in response to gestures inaccordance with embodiments;

FIG. 14 illustrates a block diagram of an action in response to a touchinput; and

FIG. 15 illustrates a flowchart regarding actions taken in response togestures sensed.

DETAILED DESCRIPTION

Example embodiments will now be described more fully hereinafter withreference to the accompanying drawings; however, they may be embodied indifferent forms and should not be construed as limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey exemplary implementations to those skilled in the art.

U.S. Pat. No. 9,596,319, incorporated by reference herein in itsentirety for all purposes, generally discloses how to handle multipleuser inputs that occur simultaneously. As disclosed therein, all inputsare assumed to function in a same fashion. As disclosed therein, avirtual canvas (“Canvas”) is a virtual region that expands to greaterthan the physical area of a display region of a display, e.g., anynumber of times the physical area up to infinite. The use of the Canvasallows additional files to be accessible and can be saved, but off thedisplay region. Gestures, such as pan, zoom and pinch gestures can bemade to move and resize the scale of the Canvas, allowing the fullcanvas to be display at once or only a small section thereof. Asdisclosed therein, all inputs are assumed to function in a same fashion,i.e., allow all of the same operations to be performed, even when anoperating system (OS) is designed to operate with a single graphicaluser interface (GUI). However, this may not be practical when using aremote, e.g., non-contact, contactless, or touchless, gesture as aninput.

First, FIG. 1 will provide a brief overview of a system 100 including adisplay 110 having a touch sensor 120, e.g., overlaying an entirety ofthe display 110, that it is sensitive to touch inputs including taps andgestures, a camera 130 associated with the display 110, e.g., for videoconferencing, and a gesture sensor 140 to detect remote gestures, i.e.,non contact or touchless inputs. The system also includes a displaycomputer 150 connected to all components. The display computer mayinclude an operating system (OS), a graphic processing unit (GPU), aprimary application (PA), other applications, e.g., a web browserapplication (WBA), Power Point, designed to operate with a single ormultiple user graphical user interface (GUI), and so forth, atouchscreen (TS) driver, and a gesture sensor (GS) driver.

One or more windows W may be open on the display. The window may be afile, e.g., a video file, a word document, and so forth, a web browser,a streaming video of another display, e.g., a mobile display or a remotedisplay, and so forth.

To perform gesture sensing simultaneously while using direct touchinputs, when a user waves their hand or raises an open hand for apredetermined time, a hand cursor 160 may appear on the display 110. Thegesture sensor 140 may sense a limited set of hand gestures, e.g., thoseshown in FIGS. 2A to 2C, examples of the effects of which will bedescribed in detail below. For example, the set of hand gestures mayinclude an open hand, shown in FIG. 2A, a first or closed hand, shown inFIG. 2B, and a one-hand clap, shown in FIG. 2C. Other users may stilluser other inputs to control windows on the display 110 screen, e.g.,using a mouse and/or direct multi-touch.

When the hand cursor is not over a particular window, as shown in FIG. 1, an open hand of FIG. 2A will move the hand cursor around the display110, e.g., such that it may overlap a window as shown in FIG. 3 . Whenthe hand cursor is not over a particular window, as shown in FIG. 1 , afirst of FIG. 2B will move the entirety of the canvas on which thewindow is placed. For example, when the gesture sensor 140 senses afirst moving to the left, the entirety of the canvas may be shifted tothe left, as illustrated in FIG. 4 .

Once the hand cursor is moved over a window, as shown in FIG. 3 , afirst may be used to select and move the window, as shown in FIG. 5 ,the movement stopping when the first is opened. When an open hand ofFIG. 2A remains on that window, this may then result in a grid of reallybig buttons, on a window, e.g., still image, x-ray, and so forth, thatcan be selected pops up, e.g., as shown in FIG. 6 . Then, when the handis closed on that menu item, e.g., expand to full screen as shown inFIG. 7 or take a snapshot of the window as shown in FIG. 8 , that menuitem is performed. when the closed hand is opened and closed again onthe expand screen button on the expanded screen in FIG. 7 , the windowmay return to its original size.

If a closed hand is on the window, but does not correspond to a button,the window may be moved as discussed above with respect to FIG. 4 .Additionally, if the closed hand outside a button, but on the window,moves towards or away from the screen, the window may be zoomed in andout, respectively, as shown in FIG. 9 .

Given the viewability and maneuverability constraints associated withremote gesture input, a number of actions that may be taken using remotegesture input may be a subset of those that can be performed with amouse or touch inputs. For example, an open hand may move the windowaround the screen, while a closed hand, i.e., a fist, may select thewindow or button. A limited number of other gestures may be used tocontrol the window.

Further, other tools may be available on the display outside of thewindow to interface contactlessly with the window. For example, a lasso,an arrow, and so forth may be selected by a closed first and dragged tothe window to select a portion within a window and when let go, e.g., byopening the closed hand, may leave the selected portion within thewindow highlighted, as may be seen in FIGS. 10 to 12 . A one hand clapshown in FIG. 2C may be used to any actions taken.

Additionally, as shown in FIG. 13 , one of the buttons may change a modeof the input interface, e.g., such that the hand cursor is changed to apin cursor, to allow more precise selection within the window.

In contrast, as may be seen in FIG. 14 , when the window is activated bya touch input, many more options may be available to manipulate thewindow, e.g., including a pen to annotate, a keyboard pop up, rotate,and so forth. In other words the input interface may be different fortouch inputs than contactless inputs.

The above may be of particular interest when used in an environment inwhich physical interaction between the display and a user may beimpractical, e.g., an operating room or other circumstances requiring adegree of sterility or other types of controlled environments. A user inthe controlled environment may use gestures remotely to control thedisplay in general, while a user outside the controlled environment maymore finely control the display, i.e., has more options to manipulatewindows.

FIG. 10 illustrates a flowchart of actions taken in response to sensedgestures.

In operation S10, the present of a hand is determined. When a hand isdetected by the gesture sensor 140, in operation S20, a hand cursor isdisplayed on the display 110, e.g., near where the hand was sensed.

In operation S30, whether or not the cursor is on a window. If not,operation S40 determines if the hand is open. If not, moving the handwill move the cursor (compare FIGS. 1 and 3 ). If the hand is closed,moving the hand will move the entire canvas (compare FIGS. 1 and 4 ).

If operation S30 determines the cursor is on a window, operation S70shows buttons of for the contactless gesture input interface. Then, whenoperation S80 determines the hand is closed, operation S90 executes theselected operation, e.g., either corresponding to the button or thedefault operations where there is no button.

The methods and processes described herein may be performed by code orinstructions to be executed by a computer, processor, manager, orcontroller. Because the algorithms that form the basis of the methods(or operations of the computer, processor, or controller) are describedin detail, the code or instructions for implementing the operations ofthe method embodiments may transform the computer, processor, orcontroller into a special-purpose processor for performing the methodsdescribed herein.

Also, another embodiment may include a computer-readable medium, e.g., anon-transitory computer-readable medium, for storing the code orinstructions described above. The computer-readable medium may be avolatile or non-volatile memory or other storage device, which may beremovably or fixedly coupled to the computer, processor, or controllerwhich is to execute the code or instructions for performing the methodembodiments described herein.

Example embodiments have been disclosed herein, and although specificterms are employed, they are used and are to be interpreted in a genericand descriptive sense only and not for purpose of limitation. In someinstances, as would be apparent to one of ordinary skill in the art asof the filing of the present application, features, characteristics,and/or elements described in connection with a particular embodiment maybe used singly or in combination with features, characteristics, and/orelements described in connection with other embodiments unless otherwisespecifically indicated. Accordingly, it will be understood by those ofskill in the art that various changes in form and details may be madewithout departing from the spirit and scope of the present invention asset forth in the following claims.

What is claimed is:
 1. A graphical user interface on a display of acomputer running software to communicate with live sources and togenerate a live window associated with each live source on a canvas whenthe live source is loaded onto the canvas, the graphical user interfacecomprising: a tray region in the canvas, the tray region to display arepresentation of a live source in communication with the computer; andin response to movement of the representation of the live source out ofthe tray region, the computer is configured to open a live source windowon the canvas and render the content of the live source.
 2. Thegraphical user interface as claimed in claim 1, wherein the movementincludes dragging the representation outside of the tray region.
 3. Thegraphical user interface as claimed in claim 1, wherein the movementincludes tapping on the representation.
 4. The graphical user interfaceas claimed in claim 1, wherein the tray region includes a device regioncontaining the representation of the live source and an applicationregion containing available application icons, wherein, in response tomovement of an application icon out of the tray region, the computer isconfigured to open an application window on the canvas to run theapplication corresponding to the application icon.
 5. The graphical userinterface as claimed in claim 4, wherein the tray region furtherincludes a source region containing an image stored on the computer orstreamed from the live source, wherein, in response to movement of animage out of the tray region, the computer is configured to open theimage on the canvas.
 6. The graphical user interface as claimed in claim1, wherein the live source includes a weblink based content urlbookmark, a mobile device, a live video source, and a camera feed. 7.The graphical user interface as claimed in claim 1, wherein the trayregion further includes a source region containing an image stored onthe computer or streamed from the live source, wherein, in response tomovement of an image out of the tray region, the computer is configuredto open the image on the canvas.